Multifunction Multimedia Device

ABSTRACT

A method for interpreting messages, user-defined alert conditions, voice commands and performing an action in response is described. A method for annotating media content is described. A method for presenting additional content associated with media content identified based on a fingerprint is described. A method for identifying that an advertisement portion of media content is being played based on a fingerprint derived from the media content is described. A method of one media device recording particular media content automatically in response to another media device recording the particular media content is described. A method of concurrently playing media content on multiple devices is described. A method of publishing information associated with recording of media content is described. A method of deriving fingerprints by media devices that meet an idleness criteria is described. A method of loading, modifying, and displaying a high definition frame from a frame buffer is described. A method of recording or playing media content identified based on fingerprints is described.

PRIORITY INFORMATION

The present invention claims priority under 35 U.S.C. 119(e) ofprovisional application No. 61/242,277 filed on Sep. 14, 2009.

FIELD OF THE INVENTION

The present invention relates to a multifunction multimedia device.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Multimedia content streams may be received by a multimedia player fordisplay to a user. Furthermore, general information about multimediacontent may be received by the multimedia player for display to theuser. The multimedia content is generally presented in a fixednon-editable format. The user is able to jump to particular points inthe media content via scene selections created by the producer.Accordingly, the watching of the media content is generally passive andthe user interaction is minimal.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1A is a block diagram illustrating an example system in accordancewith an embodiment;

FIG. 1B is a block diagram illustrating an example media device inaccordance with an embodiment;

FIG. 2 illustrates a flow diagram for presenting additional content inaccordance with an embodiment.

FIG. 3 illustrates a flow diagram for determining a position in theplaying of media content in accordance with an embodiment.

FIG. 4 illustrates a flow diagram for detecting the playing of anadvertisement in accordance with an embodiment.

FIG. 5 illustrates a flow diagram for deriving a fingerprint from mediacontent in accordance with an embodiment.

FIG. 6 shows an exemplary architecture for the collection and storage offingerprints derived from media devices.

FIG. 7 illustrates a flow diagram for presenting messages in accordancewith an embodiment.

FIG. 8 illustrates a flow diagram for interpreting voice commands inaccordance with an embodiment;

FIG. 9 illustrates a flow diagram for correlating annotations with mediacontent in accordance with an embodiment;

FIG. 10 shows an exemplary system for configuring an environment inaccordance with one or more embodiments.

FIG. 11 shows a block diagram that illustrates a system upon which anembodiment of the invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Several features are described hereafter that can each be usedindependently of one another or with any combination of the otherfeatures. However, any individual feature might not address any of theproblems discussed above or might only address one of the problemsdiscussed above. Some of the problems discussed above might not be fullyaddressed by any of the features described herein. Although headings areprovided, information related to a particular heading, but not found inthe section having that heading, may also be found elsewhere in thespecification.

Example features are described according to the following outline:

-   -   1.0 FUNCTIONAL OVERVIEW    -   2.0 SYSTEM ARCHITECTURE    -   3.0 PRESENTING ADDITIONAL CONTENT BASED ON MEDIA CONTENT        FINGERPRINTS    -   4.0 DETERMINING A PLAYING POSITION BASED ON MEDIA CONTENT        FINGERPRINTS    -   5.0 PUBLISHING RECORDING OR VIEWING INFORMATION    -   6.0 DERIVING A FINGERPRINT FROM MEDIA CONTENT    -   7.0 PRESENTING UPDATES    -   8.0 INTERPRETING COMMANDS    -   9.0 CORRELATING INPUT WITH MEDIA CONTENT    -   10.0 ELICITING ANNOTATIONS BY A PERSONAL MEDIA DEVICE    -   11.0 MARKING MEDIA CONTENT    -   12.0 PUBLICATION OF MEDIA CONTENT ANNOTATIONS    -   13.0 AUTOMATICALLY GENERATED ANNOTATIONS    -   14.0 ENVIRONMENT CONFIGURATION    -   15.0 HARDWARE OVERVIEW    -   16.0 EXTENSIONS AND ALTERNATIVES

1.0 Functional Overview

In an embodiment, media content is received and presented to a user. Afingerprint derived from the media content is then used to query aserver to identify the media content. Based on the media contentidentified based on the fingerprint, additional content is obtained andpresented to the user.

In an embodiment, the additional content may include an advertisement(e.g., for a product, service, or other media content), which isselected based on the identified media content.

In an embodiment, a fingerprint is derived dynamically from the mediacontent subsequent to receiving a command to present the media content.In an embodiment, the fingerprint is derived dynamically from the mediacontent subsequent to receiving a command to present additional contentassociated with the media content being presented.

In an embodiment, a face is detected in the media content based on thefingerprint derived from the media content. A name of a personassociated with the face is determined and presented in the additionalcontent. Detecting the face and/or determining the name of the personassociated with the face may be dynamically performed in response toreceiving a user command.

In an embodiment, features (e.g., objects, structures, landscapes,locations, etc.) in media content frames may be detected based on thefingerprint derived from the media content. The features may beidentified and the identification may be presented. The features may beidentified and/or the identification presented in response to a usercommand.

In an embodiment, fingerprints may be dynamically derived concurrentlywith playing the media content. A position in the playing of the mediacontent may then be determined based on the fingerprints.

In an embodiment, additional content may be presented based on theposition in the playing of the media content. In an embodiment, theadditional content based on the position in the playing of the mediacontent may be presented in response to a user command.

In an embodiment, playing of the media content may be synchronized overmultiple devices based on the position in the playing of the mediacontent. In an embodiment, synchronization over multiple devices may beperformed by starting the playing of media content on multiple devicesat the same time, seeking to an arbitrary position of the media contenton a device or delaying the playing of media content on a device. Duringsynchronized playing of the media content on multiple devices, a commandto fast-forward, rewind, pause, stop, seek, or play on one device may beperformed on all synchronized devices. In an embodiment, a determinationmay be made that advertisements are being played based on the positionin the playing of the media content. The advertisement may be skippedover or fast-forwarded through based on the position in the playing ofthe media content. In an embodiment, a notification may be provided thatthe advertisement was played or the speed at which the advertisement wasplayed. In an embodiment, the advertisement may be selected based on theposition in the playing of the media content.

In an embodiment, the playing of an advertisement may be detected bydetermining that one or more fingerprints of the media content beingplayed are associated with an advertisement portion of the mediacontent. In an embodiment, an advertisement may be detected byidentifying the persons associated with the faces in the advertisementportion of the media content and determining that the identified personsare not actors listed for the media content. In an embodiment, theadvertisement may be enhanced with additional content pertaining to theproduct or service being advertised. In an embodiment, the advertisementmay be automatically fast-forwarded, muted, or replaced with analternate advertisement. In an embodiment, only a non-advertisementportion of the media content may be recorded by skipping over thedetected advertisement portion of the media content.

In an embodiment, a command is received to record particular mediacontent on a first device associated with a first user and theparticular media content is scheduled for recording on the first device.A notification is provided to a second device associated with a seconduser of the scheduling of the recording of the particular media contenton the first device. The second device may then schedule recording ofthe particular media content. The second device may schedule therecording of the particular media content without receiving a usercommand or subsequent to receiving a user confirmation to record theparticular media content in response to the notification.

In an embodiment, a command may be received from the second user by thesecond device to record all media content that is scheduled forrecording on first device, any one of a plurality of specified devices,or a device associated with any of a plurality of specified users.

In an embodiment, the scheduled recording of a particular media contenton multiple devices may be detected. In response to detecting that theparticular media content is scheduled for recording on multiple devices,a notification may be provided to at least one of the multiple devicesthat the particular media content is scheduled for recording on themultiple devices. The particular media content may then be synchronouslydisplayed on the multiple devices. A time may be selected by one of thedevices to synchronously play the particular media content on themultiple devices based on a user availability calendar accessiblethrough each of the devices. A time may also be suggested to receive auser confirmation for the suggested time.

In an embodiment, a command to record or play a particular media contenton a device associated with a user may be received. Responsive to thecommand, the particular media content may be recorded or played andinformation may be published in association with the user indicatingthat the user is recording or playing the particular media content. Theinformation may be automatically published to a web service for furtheraction, such as display on a web page. Responsive to the command,information associated with the particular media content may be obtainedand presented to the user. In an embodiment, a group (e.g., on a socialnetworking website) may be automatically created for users associatedwith devices playing or recording the particular media content.

In an embodiment, a media device meeting an idleness criteria may bedetected. In response to detecting an idleness criteria, media contentmay be sent to the media device. The media device may be configured toreceive a particular content stream or streams accessible via theinternet comprising the media content. The media device may derive afingerprint from the media content and send the fingerprint to afingerprint database, along with additional data pertaining to the media(such as title, synopsis, closed caption text, etc). Detecting that amedia device meets an idleness criteria may involve receiving a signalfrom the media device, the media device completing a duration of timewithout receiving a user command at the media device, or determiningthat the media content has resource availability for deriving afingerprint.

In an embodiment, concurrently with playing audio/video (AV) content, amessage is received. The message is interpreted based on messagepreferences associated with a user and the user is presented with themessage based on the message preferences. In an embodiment, one or moremessages may be filtered out based on message preferences.

In an embodiment, presenting messages includes overlaying informationassociated with the message on one or more video frames of the AVcontent being played to the user. Presenting the message may includeplaying audio information associated with the message. In an embodiment,AV content is paused or muted when messages are presented.

In an embodiment, messages are submitted by another user as audio input,textual input or graphical input. Audio input may include a voiceassociated with the sender of the message, the receiver of the message,a particular fictional character, or non-fictional character, or acombination thereof. The messages may be played exclusively to therecipient of the message.

In an embodiment, a message may be presented during a time periodspecified by a message preference. A message may be held until acommercial break during the playing of the AV content and presentedduring the commercial break. In an embodiment, a message maybe receivedfrom a message service associated with a social networking website.

In an embodiment, a user-defined alert condition is received from auser. AV content is played concurrently with monitoring for occurrenceof the user-defined alert condition and occurrence of the user-definedalert condition is detected. An alert may be presented in response todetecting occurrence of the user-defined alert condition.

In an embodiment, detecting the alert condition includes determiningthat media content determined to be of interest to a user is beingavailable on a content stream. In an embodiment, detecting the alertcondition includes determines that media content associated with userrequested information is available on a content stream. Detecting thealert condition may include receiving a notification indicatingoccurrence of the alert condition. In an embodiment, detectingoccurrence of an alert condition may include obtaining information usingoptical character recognition (OCT) and detecting occurrence of thealert condition based on the information.

In an embodiment, a voice command is received from a user and the useris identified based on voice command. The voice command is theninterpreted based on preferences associated with the identified user todetermine an action out of a plurality of actions. The action is thenperformed.

In an embodiment, a number of applicable users for the voice command isdetermined. The number of applicable users may be determined byrecognizing users based on voice input.

In an embodiment, the action based on user preferences may includeconfiguring a multimedia device or an environment, presenting messages,making a purchase, or performing another suitable action. In anembodiment, an action may be presented for user confirmation prior toperforming the action or checked to ensure that the user permission toexecute the action. In an embodiment, the voice command may beinterpreted based on the language in which the voice command wasreceived.

In an embodiment, concurrently with playing media content on amultimedia device, an annotation(s) is received from a user. Theannotation is stored in associated with the media content. In anembodiment, the annotation may include audio input, textual input,and/or graphical input. In an embodiment, the media content is played asecond time concurrently with audio input received from the user.Playing the media content the second time may involve playing only avideo portion of the media content with the audio input received fromthe user.

In an embodiment, multiple versions of annotations may be receivedduring different playbacks of the media content and each annotation maybe stored in association with the media content. The annotations may beprovided in languages different than the original language of the audioportion of the media content. Annotations may be provided withinstructions associated with intended playback. Annotations may includeautomatically generated audio based on information obtained usingoptical character recognition. In an embodiment, annotations may beanalyzed to derive annotation patterns associated with media content.Annotations may be elicited from a user and may include reviews of mediacontent. In an embodiment, user profiles may be generated based onannotations. Annotations may mark intervals or particular points in theplaying of media content, which may be used as bookmarks to resumeplaying of the media content. Intervals marked by annotations may beskipped during a subsequent playing of the media content or used tocreate a play sequence.

Although specific components are recited herein as performing the methodsteps, in other embodiments agents or mechanisms acting on behalf of thespecified components may perform the method steps. Further, althoughsome aspects of the invention are discussed with respect to componentson a system, the invention may be implemented with componentsdistributed over multiple systems. Embodiments of the invention alsoinclude any system that includes the means for performing the methodsteps described herein. Embodiments of the invention also include acomputer readable medium with instructions, which when executed, causethe method steps described herein to be performed.

2.0 System Architecture

Although a specific computer architecture is described herein, otherembodiments of the invention are applicable to any architecture that canbe used to perform the functions described herein.

FIG. 1 shows a media device A (100), a media source (110), a mediadevice N (120), a fingerprint server (130), a network device (140), anda web server (150). Each of these components are presented to clarifythe functionalities described herein and may not be necessary toimplement the invention. Furthermore, components not shown in FIG. 1 mayalso be used to perform the functionalities described herein.Functionalities described as performed by one component may instead beperformed by another component.

In an embodiment, the media source (110) generally represents anycontent source from which the media device A (100) can receive mediacontent. The media source (110) may be a broadcaster (includes abroadcasting company/service) that streams media content to media deviceA (100). The media source (110) may be a media content server from whichthe media device A (100) downloads the media content. The media source(100) may be an audio and/or video player from which the media device A(100) receives the media content being played. The media source (100)may be a computer readable storage or input medium (e.g., physicalmemory, a compact disc, or digital video disc) which the media device A(100) reads to obtain the media content. The terms streaming,broadcasting, or downloading to a device may be used interchangeablyherein and should not be construed as limiting to one particular methodof the device obtaining data. The media device A (100) may receive databy streaming, broadcasting, downloading, etc. from a broadcast service,a web server, another media device, or any suitable system with data orcontent that may accessible by the media device. Different sources maybe mentioned as different examples presented below. An exampledescribing a specific source should not be construed as limited to thatsource.

In an embodiment, the fingerprint server (130) generally represents anyserver that stores fingerprints derived from media content. Thefingerprint server (130) may be accessed by the media device A (100) todownload and/or upload fingerprints derived from media content. Thefingerprint server (130) may be managed by a content source (e.g., abroadcast service, a web service, or any other source of content) forstoring a database of fingerprints derived from media content. Thecontent source may select media content to be fingerprinted. The mediadevice A (100) may derive the fingerprint from selected media contentand provide the fingerprint to the fingerprint server (130). In anembodiment, the fingerprint server (130) may serve as a database foridentifying media content or metadata associated with media contentbased on the fingerprint derived from that media content. In anembodiment, at least a portion of the fingerprint server (130) isimplemented on one or more media devices. The media devices may beupdated continuously, periodically, or according to another suitableschedule when the fingerprint server (130) is updated.

In an embodiment, the network device (140) generally represents anycomponent that is a part of the media device A (100) or a separatedevice altogether that includes functionality to communicate over anetwork (e.g., internet, intranet, world wide web, etc.). For example,the network device (140) may be a computer communicatively coupled withthe media device A (100) or a network card in the media device A (100).The network device (140) may include functionality to publishinformation associated with the media device A (100) (e.g., mediacontent scheduled for recording on the media device A (100), mediacontent recorded on the media device A (100), media content being playedon the media device A (100), media content previously played on themedia device A (100), media content displayed on the media device A(100), user preferences/statistics collected by the media device A(100), user settings on the media device A (100), etc.). The networkdevice (140) may post the information on a website, provide theinformation in an electronic message or text message, print theinformation on a network printer, or publish the information in anyother suitable manner. The network device (140) may includefunctionality to directly provide the information to another mediadevice(s) (e.g., media device N (120)). The network device (140) mayinclude functionality to obtain information from a network. For example,the network device (140) may perform a search for metadata or any otheradditional data associated with media content and provide the searchresults to the media device A (100). Another example may involve thenetwork device (140) obtaining information associated with media contentscheduled, recorded, and/or played on media device N (120).

In an embodiment media device A (100) (or media device N (120))generally represents any media device comprising a processor andconfigured to present media content. The media device A (100) may referto a single device or any combination of devices (e.g., a receiver and atelevision set) that may be configured to present media content.Examples of the media device A (100) include one or more of: receivers,digital video recorders, digital video players, televisions, monitors,Blu-ray players, audio content players, video content players, digitalpicture frames, hand-held mobile devices, computers, printers, etc. Themedia device A (100) may present media content by playing the mediacontent (e.g., audio and/or visual media content), displaying the mediacontent (e.g., still images), printing the media content (e.g.,coupons), electronically transmitting the media content (e.g.,electronic mail), publishing the media content (e.g., on a website), orby any other suitable means. In an embodiment, media device A (100) maybe a management device which communicates with one or more other mediadevices in a system. For example, the media device A (100) may receivecommands from media device (e.g., a DVD player, a remote, a joystick,etc.) and communicate the command to another media device (e.g., amonitor, a receiver, etc.). In an embodiment, the media device A (100)may represent any apparatus with one or more subsystems configured toperform the functions described herein.

In an embodiment, the media device A (100) may include functionality toderive fingerprints from media content. For example, the media device A(100) may derive a fingerprint from media content recorded on associatedmemory or stored in any other accessible location (e.g., an externalhard drive, a DVD, etc.). The media device A (100) may also derive afingerprint from media content available on a content stream. Mediacontent that is available on a content stream includes any media contentthat is accessible by the media device A (100). For example, contentavailable on a content stream may include content being broadcasted by abroadcast service, content available for download from a web server,peer device, or another system, or content that is otherwise accessibleby the media device A (100). In an embodiment, the media device A (100)may include functionality to obtain media content being displayed anddynamically derive fingerprints from the media content being displayedor media content stored on the media device. In an embodiment, the mediadevice A (100) may include the processing and storage capabilities todecompress media content (e.g., video frames), modify and/or edit mediacontent, and compress media content.

In an embodiment, the media device A (100) may include functionality tomimic another media device(s) (e.g., media device N (120)) by recording,or playing the same media content as another media device. For example,the media device A (100) may include functionality to receivenotifications of media content being recorded on media device N (120)and obtain the same media content from a content source. The mediadevice A may automatically record the media content or provide thenotification to a user and record the media content in response to auser command.

FIG. 1B illustrates an example block diagram of a media device inaccordance with one or more embodiments. As shown in FIG. 1B, the mediadevice (100) may include multiple components such as a memory system(155), a disk (160), a central processing unit (CPU) (165), a displaysub-system (170), an audio/video input (175), a tuner (180), a networkmodule (190), peripherals unit (195), text/audio convertor (167), and/orother components necessary to perform the functionality describedherein.

In an embodiment, the audio/video input (175) may correspond to anycomponent that includes functionality to receive audio and/or videoinput (e.g., HDMI 176, DVI 177, Analog 178) from an external source. Forexample, the audio/video input (175) may be a DisplayPort or a highdefinition multimedia interface (HDMI) that can receive input fromdifferent devices. The audio/video input (175) may receive input from aset-top box, a Blu-ray disc player, a personal computer, a video gameconsole, an audio/video receiver, a compact disk player, an enhancedversatile disc player, a high definition optical disc, a holographicversatile disc, a laser disc, mini disc, a disc film, a RAM disc, avinyl disc, a floppy disk, a hard drive disk, etc. The media device(100) may include multiple audio/video inputs (175).

In an embodiment, the tuner (180) generally represents any inputcomponent that can receive a content stream (e.g., through cable,satellite, internet, network, or terrestrial antenna). The tuner (180)may allow one or more received frequencies while filtering out others(e.g., by using electronic resonance). A television tuner may convert anRF television transmission into audio and video signals which can befurther processed to produce sound and/or an image.

In an embodiment, input may also be received from a network module(190). A network module (190) generally represents any input componentthat can receive information over a network (e.g., internet, intranet,world wide web, etc.). Examples of a network module (190) include anetwork card, network adapter, network interface controller (NIC),network interface card, Local Area Network adapter, Ethernet networkcard, and/or any other component that can receive information over anetwork. The network module (190) may also be used to directly connectwith another device (e.g., a media device, a computer, a secondarystorage device, etc.).

In an embodiment, input may be received by the media device (100) fromany communicatively coupled device through wired and/or wirelesscommunication segments. Input received by the media device (100) may bestored to the memory system (155) or disk (160). The memory system (155)may include one or more different types of physical memory to storedata. For example, one or more memory buffers (e.g., an HD frame buffer)in the memory system (155) may include storage capacity to load one ormore uncompressed high definition (HD) video frames for editing and/orfingerprinting. The memory system (155) may also store frames in acompressed form (e.g., MPEG2, MPEG4, or any other suitable format),where the frames are then uncompressed into the frame buffer formodification, fingerprinting, replacement, and/or display. The memorysystem (155) may include FLASH memory, DRAM memory, EEPROM, traditionalrotating disk drives, etc. The disk (160) generally represents secondarystorage accessible by the media device (100).

In an embodiment, central processing unit (165) may includefunctionality to perform the functions described herein using any inputreceived by the media device (100). For example, the central processingunit (165) may be used to dynamically derive fingerprints from mediacontent frames stored in the memory system (155). The central processingunit (165) may be configured to mark or identify media content orportions of media content based on tags, hash values, fingerprints, timestamp, or other suitable information associated with the media content.The central processing unit (165) may be used to modify media content(e.g., scale a video frame), analyze media content, decompress mediacontent, compress media content, etc. A video frame (e.g., an HD videoframe) stored in a frame buffer may be modified dynamically by thecentral processing unit (165) to overlay additional content (e.g.,information about the frame, program info, a chat message, systemmessage, web content, pictures, an electronic programming guide, or anyother suitable content) on top of the video frame, manipulate the videoframe (e.g., stretching, rotation, shrinking, etc.), or replace thevideo frame in real time. Accordingly, an electronic programming guide,advertisement information that is dynamically selected, media contentinformation, or any other text/graphics may be written onto a videoframe stored in a frame buffer to superimpose the additional content ontop of the stored video frame. The central processing unit (165) may beused for processing communication with any of the input and/or outputdevices associated with the media device (100). For example, a videoframe which is dynamically modified in real time may subsequently betransmitted for display. The central processing unit (165) may be usedto communicate with other media devices to perform functions related tosynchronization, or publication of data.

In an embodiment, the text/audio convertor (167) generally representsany software and/or hardware for converting text to audio and/or forconverting audio to text. For example, the text/audio convertor mayinclude functionality to convert text corresponding to closed captioneddata to an audio file. The audio file may be based on a computerizedvoice, or may be trained for using the voice of a user, a fictional ornon-fictional character, etc. In an embodiment, the automaticallygenerated voice used for a particular message may be the voice of a usergenerating the message. The text/audio convertor may includefunctionality to switch languages when converting from voice to text orfrom text to voice. For example, audio input in French may be convertedto a text message in English.

In an embodiment, the peripherals unit (195) generally represents inputand/or output for any peripherals that are communicatively coupled withthe media device (100) (e.g., via USB, External Serial AdvancedTechnology Attachment (eSATA), Parallel ATA, Serial ATA, Bluetooth,infrared, etc.). Examples of peripherals may include remote controldevices, USB drives, a keyboard, a mouse, a microphone, and voicerecognition devices that can be used to operate the media device (100).In an embodiment, multiple microphones may be used to detect sound,identify user location, etc. In an embodiment, a microphone may be apart of a media device (100) or other device (e.g., a remote control)that is communicatively coupled with the media device (100). In anembodiment, the media device (100) may include functionality to identifymedia content being played (e.g., a particular program, or a position ina particular program) when audio input is received (e.g., via amicrophone) from a user.

In an embodiment, the display sub-system (170) generally represents anysoftware and/or device that includes functionality to output (e.g.,Video Out to Display 171) and/or actually display one or more images.Examples of display devices include a kiosk, a hand held device, acomputer screen, a monitor, a television, etc. The display devices mayuse different types of screens such as a liquid crystal display, cathoderay tube, a projector, a plasma screen, etc. The output from the mediadevice (100) may be specially for formatted for the type of displaydevice being used, the size of the display device, resolution (e.g.,720i, 720p, 1080i, 1080p, or other suitable resolution), etc.

3.0 Presenting Additional Content Based on Media Content Fingerprints

FIG. 2 illustrates a flow diagram for presenting additional content inaccordance with an embodiment. One or more of the steps described belowmay be omitted, repeated, and/or performed in a different order.Accordingly, the specific arrangement of steps shown in FIG. 2 shouldnot be construed as limiting the scope of the invention.

Initially, a command is received to present media content in accordancewith an embodiment (Step 202). The received command may be entered by auser via a keyboard or remote control. The command may be a selection inthe electronic programming guide (EPG) by a user for the recordingand/or playing of the media content. The command may a channel selectionentered by a user. The command may be a request to display a slide showof pictures. The command may be to play an audio file. The command maybe a request to play a movie (e.g., a command for a blu-ray player). Inan embodiment, receiving the command to present media content mayinclude a user entering the title of media content in a search field ona user interface. In an embodiment, media content is presented (Step204). Presenting the media content may include playing audio and/orvisual media content (e.g., video content), displaying or printingimages, etc. Presenting the media content may also involve overlayingthe media content over other media content also being presented.

In an embodiment, a fingerprint is derived from the media content (Step206). An example of deriving a fingerprint from media content includesprojecting intensity values of one or more video frames onto a set ofprojection vectors and obtaining a set of projected values. Afingerprint bit may then be computed based on each of the projectedvalues and concatenated to compute the fingerprint for the mediacontent. Another example may include applying a mathematical function toa spectrogram of an audio file. Other fingerprint derivation techniquesmay also be used to derive a fingerprint from media content inaccordance with one or more embodiments. In an embodiment, thefingerprint is derived from media content dynamically as the mediacontent is being played. For example, media content being received froma content source may concurrently be played and fingerprinted. Thefingerprint may be derived for media content recognition, e.g.,identifying the particular program, movie, etc. Media streams containing3-Dimensional video may also be fingerprinted. In an embodiment,fingerprinting 3-Dimensional video may involve selecting fingerprintportions of the 3-Dimensional video. For example, near objects (e.g.,objects that appear closer when watching the 3-Dimensional video) in the3-Dimensional video stream may be selected for fingerprinting in orderto recognize a face or structure. The near objects may be selected basedon a field of depth tag associated with objects or by the relative sizeof objects compared to other objects.

In an embodiment, a command to present additional content associatedwith the media content being presented, is received (Step 208). Acommand may be received to identify generic additional content (e.g.,any feature in the media content). For example, information of the mediacontent being played such as the plot synopsis of a movie, the actors ina movie, the year the movie was made, a time duration associated withthe particular media content, a director or producer of the movie, agenre of the movie, etc. In an embodiment, specific information may berequested. For example, a command requesting the geographic location inthe world of the current scene being played. Another example may involvea command requesting an identification of the people in a current scenebeing displayed. Another example may involve a request for the year andmodel of a car in a scene of the movie. Another example may involve arequest to save or publish information about the content, including atimestamp, offset from beginning, and other contextual data, for lateruse or reference. Accordingly, the specific information requests mayinclude identification of places, objects, or people in a scene of themedia content.

The additional content requested by the user may not be available whenthe command for the additional content is received. Accordingly, theadditional information is dynamically identified (Step 210), afterreceiving the command, based on a fingerprint of the media content. Forexample, the fingerprint derived from the media content may be used toquery a web server and receive identification of the object, place, orperson in a scene that matches the fingerprint. The fingerprint may alsobe used to identify the media content being played to obtain themetadata already associated with the media content. In an embodiment, afingerprint may be dynamically derived from the media content afterreceiving the command to present additional information.

In an embodiment, the additional content is presented (Step 212).Presenting the additional content may include overlaying the additionalcontent on top of the media content being presented to the user.Presenting the additional content may also include overlaying theadditional content on portions of the frame displaced by scaling,cropping, or otherwise altering the original content. To overlay theadditional content on top of the original or altered media content,uncompressed HD frame(s) may be loaded into a frame buffer and theadditional data may be written into the same frame buffer, therebyoverlaying original frame information with the additional data. Theadditional information may be related to the media content being played,EPG display data, channel indicator in a banner display format asdescribed in U.S. Pat. No. 6,642,939, owned by the applicant andincorporated herein by reference, program synopsis, etc. For example, ina movie, a geographical location of the scene may be displayed on thescreen concurrently with the scene. In another example, a field maydisplay the names of current actors in a scene at any given time. Avisual indication linking the name of an object, place, person, etc.with the object, place, person on screen may be displayed. For example,a line between a car in the scene and identifying information about thecar. The additional content may also provide links to advertisers,businesses, etc. about a displayed image. For example, additionalinformation about a car displayed on the screen may include identifyinginformation about the car, a name of a car dealership that sells thecar, a link to a car dealership that sells the car, pricing informationassociated with the car, safety information associated with the car, orany other information directly or tangentially related to the identifiedcar. Another example may involve presenting information about contentavailable on a content stream (e.g., received from a broadcast serviceor received from a web server). The content itself may be overlaid onthe frame, or a link with a description may be overlaid on the frame,where the link can be selected through user input. The additionalcontent may be presented as closed caption data. In another example,subtitles in a user-selected language may be overlaid on top of thecontent, such as a movie or TV show. The subtitles may be derived byvarious methods including download from an existing database of subtitlefiles, or real-time computational translation of closed captioning textfrom the original content. Another example may involve synchronizedoverlay of lyrics on top of a music video or concert performance. Thesystem may perform this operation for several frames or until the userinstructs it to remove the overlay. At that point, the system maydiscontinue writing the additional information into the frame buffer. Inone embodiment, audio content may replace or overlay the audio from theoriginal content. One example may involve replacing the audio stream ofa national broadcast of a national football game with the audio streamof the local radio announcer. One example may involve a real-time mix ofthe audio from the original media with additional audio, such as actor'scommentary on a scene. This example may involve alteration of theoriginal and additional audio, such as amplification.

4.0 Determining a Playing Position Based on Media Content Fingerprints

FIG. 3 illustrates a flow diagram for determining a position in theplaying of media content in accordance with an embodiment. One or moreof the steps described below may be omitted, repeated, and/or performedin a different order. Accordingly, the specific arrangement of stepsshown in FIG. 3 should not be construed as limiting the scope of theinvention.

Initially, a command is received to present media content (Step 302) andthe media content is presented (Step 304) in accordance with anembodiment. Step 302 and Step 304 are essentially the same as Step 202and Step 204 described above.

In an embodiment, a fingerprint is derived from the media content beingplayed (Step 306) to determine the position in the playing of the mediacontent on a first device (Step 308). For example, as a media devicereceives media content in a content stream (or from any other source),the media device may display the media content and derive fingerprintsfrom the specific frames being displayed. The media device may alsoderive fingerprints from every nth frame, from iframes, or based on anyother frame selection mechanism. A content fingerprint derived from oneor more frames may then be compared to a database of fingerprints toidentify a database fingerprint that matches the frame fingerprint. Thedatabase of fingerprints may be locally implemented on the media deviceitself or on a server communicatively coupled with the media device. Thematch between the content fingerprint and the database fingerprint maybe an exact match or the two fingerprints may meet a similaritythreshold (e.g., at least a threshold number of signature bits in thefingerprint match). Once a match is identified in the database, metadatathat is stored in association with the database fingerprint is obtained.The metadata may include a position in the media content. For example,the metadata may indicate that the fingerprint corresponds to the kthframe of n total frames in the media content. Based on this positioninformation and/or the number of frames per second, a position in theplaying of the media content may be determined. The metadata may alsoexplicitly indicate the position. For example, the metadata may indicatethat the fingerprint corresponds to a playing position at 35 minutes and3 seconds from the start of the media content.

Based on the position in the playing of the media content on the firstdevice, a second device may be synchronized with the first device byplaying the same media content on the second device concurrently, inaccordance with one or more embodiments. (Step 310). Once a position ofthe playing of the media content is determined for the first device, theplaying of the media content on the second device may be started at thatposition. If the media content is already being played on the seconddevice, the playing of the media content on the second device may bestopped and restarted at that position. Alternatively, the playing ofthe media content on the second device may be fast forwarded or rewoundto that position.

In an embodiment, the viewing of a live broadcast or stored program maybe synchronized using a buffer incorporated in media devices. Forexample, the content received in the content stream may be stored onmultiple devices as they are received. Thereafter, the devices maycommunicate to synchronously initiate the playing of the media content,the pausing of media content, the fast forwarding of media content, andthe rewinding of media content. A large buffer that can store the entiremedia content may be used in an embodiment. Alternatively, a smallerbuffer can be used and video frames may be deleted as they are displayedand replaced with new video frames received in a content stream.Synchronized playing of a live broadcast or stored program may involveplaying a particular frame stored in a memory buffer at a particulartime to obtain frame level synchronization. For example, two devices mayexchange information that indicates at which second a particular framestored in memory is to be played and a rate at which future frames areto played. Accordingly, based on the same start time, the frames may bedisplayed on different media devices at the exact same time orapproximately the same time. Furthermore, additional frame/timecombinations may be determined to ensure that the synchronization ismaintained. When media devices are being used in different time zones,the times may be adjusted to account for the time difference. Forexample, Greenwich Mean Time (GMT) may be used across all media devicesfor synchronized playing of media content.

In an embodiment, after synchronization of multiple devices playing thesame media content, the synchronization may be maintained. In order tomaintain synchronization any play-function (e.g., stop, fast-forward,rewind, play, pause, etc.) received on one device may be performed onboth devices (Step 312).

In an embodiment, the playing of an advertisement may be detected basedon the position in the playing of the media content (Step 314). Forexample, media content available on a content stream may include atelevision show and advertisements interspersed at various times duringthe television show. The composition information of the media contentmay indicate that the television show is played for twenty-five minutes,followed by five minutes of advertisements, followed by anothertwenty-five minutes of the television show and followed again by anotherfive minutes of advertisements. Accordingly, if the position of theplaying of the media content is determined to be twenty minutes from thestart, the television show is being played. However, if the position ofthe playing of the media content is determined to be twenty-sevenminutes from the start, an advertisement is being played.

In an embodiment, the playing of an advertisement may be detectedwithout determining the position in the playing of the media content.For example, if the media content includes a television show andadvertisements interspersed between the television show, advertisementsmay be detected based on the fingerprints derived from the media contentcurrently being played. The fingerprints derived from the media contentcurrently being played may be compared to the fingerprints derived onlyfrom the television show or fingerprints derived only from theadvertisement. Based on the comparison, the media content concurrentlybeing played may be determined to be a portion of the television show ora portion of the advertisement.

In an embodiment, the playing of an advertisement may be detected basedon the elements present in the media content. For example, based on thefingerprints derived from the media content being played, faces ofactors within the media content may be recognized. The names of theactors may then be compared with the names of actors that are listed asactors in the television show. If the actors detected in the mediacontent being played match the actors listed as actors in the televisionshow, then the television show is being played. Alternatively, if theactors detected in the media content being played do not match theactors listed as actors in the television show, then an advertisement isbeing played. In an embodiment, a time window may be used for detectionof known actors in a television show, where at least one actor listed asan actor in the television show must be detected within the time windowto conclude that the television show is being played.

In response to determining that an advertisement is being played, manydifferent actions may be performed in accordance with one or moreembodiments. In an embodiment, advertisements may be autofast-forwarded. For example, as soon as the playing of an advertisementis detected, an automatic fast-forwarding function may be applied to theplaying of the media content until the playing of the advertisement iscompleted (e.g., when playing of a television program is detected againbased on a fingerprint). Similarly, advertisements may also beauto-muted, where an un-muting function is selected in response todetecting the completion of the advertisement.

In an embodiment, if the media content is being recorded, anadvertisement may automatically be skipped over for the recording. Forexample, in the recording of a movie being received from a contentsource, the non-advertisement portions (e.g., movie portions) of themedia content may be recorded while the advertisement portions of themedia content may be skipped for the recording.

In an embodiment, alternate advertisements may be displayed. Whenreceiving and displaying a content stream, detected advertisementportions of the content stream may be replaced with alternateadvertisements. For example, a media device at a sports bar may beprogrammed to display drink specials instead of the advertisementsreceived in a content stream. Alternatively, advertisements from localvendors, which are stored in memory or streamed from a server, may bedisplayed instead of advertisements received in the content stream. Theadvertisements may be selected based on the media content. For example,if during the playing of a sporting event, advertisements directedtoward men may be selected.

In an embodiment, the advertisement may be augmented with additionalcontent related to the advertisement. When receiving a content stream,detected advertisement portions of the content stream may be scaled,cropped, or otherwise altered, and the displaced empty space can beprogrammatically populated by additional content. For example, anadvertisement for a movie opening in theaters soon can be augmented withshow times at theaters in a 15-mile vicinity of the device. The user mayalso be presented with one or more interactive functions related to theadditional content, such as the option to store information about theadvertised movie, including the selected local theater and show time, tobe used in future presentation, reference, ticket purchase, or otherrelated activity. In another example, the advertisement may be augmentedwith games, quizzes, polls, video, and audio related to theadvertisement. In an embodiment, the advertisement may be augmented withinformation about actions taken by the user's social network connectionsrelated to the advertisement. For example, an advertisement for adigital camera may be augmented by photos of the user's friends takenwith the same digital camera. In another example, an advertisement for amovie recently released on DVD may be augmented with friends' ratingsand reviews of that movie.

In an embodiment, the advertisement may be augmented with additionalcontent not related to the advertisement. When receiving a contentstream, detected advertisement portions of the content stream may bescaled, cropped, or otherwise altered, and the displaced empty space canbe programmatically populated by additional content. In one embodiment,the user may direct the system to use portions of the display duringadvertisements to display personalized content. In one example, thepersonalized content may include the latest scores and statistics fromthe user's favorite sports teams. In another example, the content mayinclude all or some of the user's latest received messages, such asemail, SMS, instant messages, social network notifications, and voicemails. In another example, the user may be presented with informationabout additional content related to the content interrupted by theadvertisement. In another example, the user may be presented with thechance to take his turn in a previously started game. In an embodiment,the user may also be presented with one or more interactive functionsrelated to the additional content, such as the option to storeinformation about the content to be used in future presentation,reference, or other related activity. In an example, the user may chooseto respond to an SMS, email, voice mail, or instant message using akeyboard or microphone.

In an embodiment, a notification of the playing of an advertisement by amedia device may be provided to an interested party (e.g., a vendor orbroadcaster). For example, if a vendor advertisement is played on amedia device, a content source may be informed that the vendoradvertisement was in fact played. Furthermore, if a vendor advertisementwas fast forwarded through, the content source may be informed that thevendor advertisement was fast forwarded through. This information may beprovided to the vendor in order for the vendor to determine theeffectiveness of the advertisement. Additional information includingwhether the advertisement was played as a part of a previously storedrecording or played directly upon receiving from the content source maybe provided to an interested party.

In an embodiment, cumulative statistics of a user may also be gatheredbased on advertisement detection. For example, particular types ofadvertisements or media content viewed by a user may be documented todetermine user interests. These user interests may be provided to avendor, stored on a server, published on an interactive webpageassociated with the user, or otherwise presented. Anonymous informationof a plurality of users may be collected to create reports based on userviewing or input. U.S. patent application Ser. No. 10/189,989, owned bythe Applicant and incorporated herein by reference, describes suchapproaches.

5.0 Publishing Recording or Viewing Information

FIG. 4 illustrates a flow diagram for detecting the playing of anadvertisement in accordance with an embodiment. One or more of the stepsdescribed below may be omitted, repeated, and/or performed in adifferent order. Accordingly, the specific arrangement of steps shown inFIG. 4 should not be construed as limiting the scope of the invention.

In an embodiment, a command is received to view or record media contenton a first device associated with a first user (Step 402). The commandto view or record media content may be received by a selection in anelectronic programming guide (EPG). The command may be for a singlerecording of media content (e.g., a movie, a sports event, or aparticular television show) or a series recording of media content(e.g., multiple episodes of a television show). A command may bereceived to play a media content file that is locally stored on memory(e.g., a DVD player may receive a command to play a DVD, a digital videorecorder may receive a command to play a stored recording). In anembodiment, a single media device may receive all such commands andinstruct the other devices (e.g., a DVD player, a blu-ray player)accordingly.

The viewing or recording of media content on the first device ispublished in accordance with an embodiment (Step 404). Publishing theviewing or recording of media content may be user specific. For example,the viewing or recording of media content may be posted on a webpage(e.g., a user webpage on a networking website such as MySpace®, orFacebook®) (MySpace® is a registered trademark of MySpace, Inc., BeverlyHills, Calif. and Facebook® is a registered trademark of Facebook, Inc.,Palo Alto, Calif.) associated with a user, a posting on a group page(e.g., a webpage designated for a group) may be emailed to other users,may be provided in a text message, or may be published in any othermanner. In an embodiment, all the viewing or recording by a user may beautomatically emailed to a list of other users that have chosen toreceive messages from the user (e.g., using Twitter®, Twitter® is aregistered trademark of Twitter, Inc., San Francisco, Calif.).Publishing the viewing or recording of media content may also include afee associated with the media content. For example, if the user selectsa pay per view movie, the cost of the movie may also be published. In anembodiment, publishing the viewing or recording of media content mayinvolve publishing the name of a user (or username associated with theuser) on a publication associated with the media content. For example,all the users that have viewed a particular media content may bepublished on a single web page associated with a social networkingwebsite. Any users that have responded (e.g., “like”, “thumbs up”,“share”, etc.) to a posting related to the particular media content,which indicates the user has viewed the particular media content, may bepublished on the single web page.

In an embodiment, responsive to receiving a command to record mediacontent on the first device associated with a first user, the mediacontent is recorded on the first device and a second device associatedwith a second user (Step 506). For example, the first device may notifythe second device of the scheduled recording of media content and thesecond device may auto-record the media content. In another example, inresponse to the notification from the first device, the second devicemay prompt a second user for recording of the media content. The seconddevice may then record the media content subsequent to receiving a usercommand to record the media content. In an embodiment, the recording ofthe media content on the second device may be subsequent to thepublication (e.g., on a website) of recording on the first device, asdescribed above. For example, a second user may select a link on awebsite associated with the publication of recording the media contenton the first device, to record the media content on the second deviceassociated with the second user. In an embodiment, a media device may beconfigured to mimic another media device by recording all programsrecorded by the other media device.

The recording of the same media content on multiple devices may bedetected in accordance with an embodiment (Step 408). For example,different users within a user group may each schedule the recording ofthe same media content on their respective media devices. The scheduledrecordings of each media device associated with the users within thegroup may be collected and compared (e.g., by a server, a service, orone of the media devices) to detect any overlapping scheduledrecordings. In an embodiment, the already recorded media content on amedia device may be compared to the already recorded media content onanother media content or to scheduled recordings on another mediacontent.

In an embodiment, a media device may be configured to automaticallyschedule recordings of any media content that is scheduled for recordingby another specified media device. Accordingly, a media device may beconfigured to mimic another media device identified by a deviceidentification number. The media device may also be configured to mimicany device associated with a specified user. For example, a first usermay determine that a second user has a great selection of new shows orprograms based on the postings of the second user on a social networkingwebsite. The first user may then choose to mimic the television watchinghabits of the second user by submitting a mimicking request with theidentification number of the media device associated with the seconduser or a name of the second user. Alternatively, the first user mayindicate the preference on the social networking website. The socialnetworking website may then communicate the identification of the firstuser and the second user to a content source, which configures the mediadevice associated with the first user to record the same shows asrecorded by the media device associated with the second user.

In an embodiment, each media device may be configured to access adatabase of media device recording schedules (e.g., on a server,provided by a third party service, etc.). A user may access thisdatabase using their own media device and mimic the recordings ofanother media device that is referenced by the name or identification ofa specific user. For example, a user may select specific shows that arealso recorded by another user. In an embodiment, the user may be able toaccess other recording related statistics to select shows for viewing orrecording. For example, a media device recording database may indicatethe most popular shows based on future scheduled recordings, based onrecordings already completed, or based on a number of users that watchedthe shows as they were made available on the content stream.

A time for playing the media content concurrently on multiple devicesmay be scheduled in accordance with an embodiment (Step 410). The timefor playing the media content may be selected automatically or may beselected based on user input from one or more users. For example, allusers associated with media devices that are scheduled for recording (orhave already recorded) particular media content may be notified of theoverlapping selection and one user may select the time for concurrentviewing of the media content by all the users using their respectivemedia devices. In another example, each media device may access a useravailability calendar to determine the available viewing times for arespective user. Thereafter, a synchronous viewing of a show may bescheduled in the calendar such that all the users (or most of the users)are available.

The viewers/recorders of the same media content may be automaticallyenrolled into a group associated with the media content in accordancewith an embodiment (Step 412). For example, all the viewers and/orrecorders of a specific movie may be automatically enrolled into asocial networking group associated with the movie, in response to eachrecording/viewing the movie. The auto-enrollment group may be used byusers as a forum to discuss the media content, find other users withsimilar viewing preferences, schedule a viewing time for similarrecordings, or for any other suitable purpose. A discussion forum may beinitiated for two or more users associated with multiple devices thatare synchronously playing media content. The discussion forum may beinitiated by the media device inviting a user to join an instantmessaging chat (e.g., Yahoo!® Instant Messaging, Google® Chat, AIM®,Twitter®, etc.) (Yahoo!® is a registered trademark of Yahoo!, Inc.,Sunnyvale, Calif.|Google® is a registered trademark of Google, Inc.,Mountain View, Calif.|AIM® is a registered trademark of AOL LLC, Dulles,Va.|Twitter® is a registered trademark of Twitter, Inc., San Francisco,Calif.), video chat (e.g., Skype®, Skype® is a registered trademark ofSkype Limited Corp., Dublin, Ireland), a website thread, or anelectronic messaging (email) thread. The discussion forum may includetwo users or any number of users. The discussion forum may be initiatedfor users that are already known to be connected. For example, thediscussion forum may be initiated if users are friends on a socialnetworking website. In an embodiment, the discussion forum may becreated to introduce vendors to potential clients. For example, duringthe playing of a football game, an invitation may be presented to chatwith a vendor of football game tickets. In an embodiment, the discussionforum may be implemented as a dating portal. For example, men and womenin the same geographical area that are subscribed to a dating server,who are watching the same show may be invited to a chat by the mediadevice. Another example involves an activity portal. For example, amedia device may be configured to invite viewers of a cooking channelshow to cook together, or a media device may configured to inviteviewers of a travel channel show to travel to a featured destinationtogether. A media device may be configured to communicate, as describedabove, with any other computing device (e.g., another media device or apersonal computer).

6.0 Deriving a Fingerprint from Media Content

FIG. 5 illustrates a flow diagram for deriving a fingerprint from mediacontent in accordance with an embodiment. One or more of the stepsdescribed below may be omitted, repeated, and/or performed in adifferent order. Accordingly, the specific arrangement of steps shown inFIG. 5 should not be construed as limiting the scope of the invention.

In an embodiment, a media device is monitored to determine that themedia device meets an idleness criteria (Step 502). An idleness criteriamay be based on non-use of a media device or component, or a usagepercentage (e.g., a percentage related to available bandwidth of thetotal bandwidth or a percentage related to available processing power ofthe total processing power). The media device may be self monitored ormonitored by a server. Monitoring the media device for the idlenesscriteria may involve detecting completion of a period of time withoutreceiving a user command. Monitoring the media device for the idlenesscriteria may involve detecting availability of resources needed toreceive media content and/or derive a fingerprint from the mediacontent. Monitoring the media device may include separately monitoringdifferent components of a media device. For example, if a user iswatching a stored recording on the media device and not recording anyadditional content being streamed to the media device, the tuner may beidle. Based on this information, a determination may be made that thetuner meets an idleness criteria. Accordingly, different components ofthe media device may be associated with separate idleness criteria. Inanother example, components necessary for deriving a fingerprint frommedia content may meet an idleness criteria.

In an embodiment, the media device receives media content from a contentsource for the purpose of deriving a fingerprint from the media content(Step 504). The media device may receive media content in response toalerting a content source that the media device (or components withinthe media device) meet an idleness criteria. In an embodiment, thecontent source may automatically detect whether a media device meets anidleness criteria. For example, the content source may determine thatthe media device has not requested to view any particular media content(e.g., broadcast content, web content, etc.). Therefore, the tuner mostlikely has bandwidth to download media content. In an embodiment, mediadevices may include the functionality to receive multiple contentstreams. In this embodiment, the content source may determine how manycontent streams are being received by the media device. Based on theknown configuration and/or functionality of the media device, thecontent source may determine the tuner's available bandwidth forreceiving additional media content. Once the idleness criteria is met,the content source may download a particular media content for the mediadevice to generate a fingerprint.

In an embodiment, the content source may build a database offingerprints for media content by dividing out the media content to bebroadcasted among multiple media devices that meet the idlenesscriteria. For example, if five thousand devices meet the idlenesscriteria and two thousand unique media content files are to befingerprinted, the content source might transmit four unique mediacontent files to each of the five thousand media devices for generatingrespective fingerprints from the media devices. In an embodiment, thecontent source may send each unique media content file to two or moremedia devices in case there is an error with the fingerprint derivedfrom media device, or if the media device is interrupted while derivingthe fingerprint. The content source may also direct a media device tofingerprint content which has already been downloaded to the mediadevice (e.g., based on user command). In an embodiment, a user mayresume utilizing the media device and thereby prevent or stop the mediadevice from deriving a fingerprint. In an embodiment, the content sourcemay prompt the user to request permission for using the media devicewhen an idleness criteria is met before downloading media content ontothe media device. The content source may also offer incentives such ascredits to watch pay-per-view movies if the user allows the contentsource to use the media device to perform and/or execute particularfunctions (e.g., deriving fingerprints).

In an embodiment, a fingerprint is derived from media content by themedia device (Step 506). Any technique may be used to derive afingerprint from media content. One example is to derive a fingerprintfrom a video frame based on the intensity values of pixels within thevideo frame. A function (e.g., that is downloaded onto the media device)may be applied to each of the intensity values and thereafter based onthe result, a signature bit (e.g., ‘0’ or ‘1’) may be assigned for thethat intensity value. A similar technique may be used for audiofingerprinting by applying the method to spectrograms created from audiodata.

The fingerprint may be derived by the media device based on specificinstructions from the content source. For example, fingerprints may bederived from all video frames of a particular media content file.Alternatively, the fingerprint may be derived for every nth frame orevery iFrame received by the media device. In an embodiment, specificframes to be fingerprinted may be tagged. Tagging techniques aredescribed in application Ser. No. 09/665,921, application Ser. No.11/473,990, and application Ser. No. 11/473,543, all of which are ownedby the Applicant, and herein incorporated by reference. Once a mediadevice receives a frame that is tagged, the media device may thendecompress the frame, analyze the frame, and derive a fingerprint fromthe frame. The video frame fingerprints may be categorized by the mediadevice according to the media content (e.g., by media content name,episode number, etc.).

In an embodiment, the media device may derive fingerprints for mediacontent that is being watched by a user. For example, a user may selecta particular show on an electronic programming guide displayed by amedia device. The media device may then request the content stream, fromthe content source, that includes the particular show. As an optionalstep, the source may indicate whether a fingerprint is needed for theparticular show requested by the media device. The indication may be aflag in the data received by the media device. If the particular showneeds to be fingerprinted as indicated by the flag, the media device maydecompress the corresponding video frames, load the decompressed videoframes into memory and analyze the video frames to derive a fingerprintfrom the video frames. In an embodiment, the user may change the channelmid-way through the playing of the media content being fingerprinted. Asa result the tuner may be forced to receive a different content stream.In this case, the media device may have derived fingerprints for only aportion of the media content. The media device may generate metadataindicating the start position and end position in the playing of themedia content for which the fingerprint has been derived.

In an embodiment, the media device may then upload the fingerprintderived from the media content (or from a portion of the media content)to a fingerprint server in accordance with an embodiment (Step 508).Thus, a fingerprint database may be built by multiple media devices eachuploading fingerprints for media content. Fingerprints received for onlya portion of the media content may be combined with other fingerprintsfrom the same media content to generate a complete fingerprint. Forexample, if one media device generates and uploads fingerprints forvideo frames in the first half of a program and a second media devicegenerates and uploads fingerprints for a second half of the sameprogram, then the two fingerprints received from the two devices may becombined to obtain fingerprints for all the video frames of the program.

An exemplary architecture for the collection and storage of fingerprintsderived from media devices, in accordance with one or more embodimentsis shown in FIG. 6. The fingerprint management engine (604) generallyrepresents any hardware and/or software that may be configured to obtainfingerprints derived by media devices (e.g., media device A (606), mediadevice B (608), media device C (610), media device N (620), etc.). Thefingerprint management engine (600) may be implemented by a contentsource or other system/service that includes functionality to obtainfingerprints derived by the media devices. The fingerprint managementengine (604) may obtain fingerprints for media content already receivedby the media device (e.g., in response to user selection of the mediacontent or content stream which includes the media content). Thefingerprint management engine (604) may transmit media content to amedia device specifically for the purpose of deriving a fingerprint. Thefingerprint management engine (604) may transmit media content to amedia device for fingerprinting in response to detecting that the mediadevice is idle. In an embodiment, the fingerprint management engine(604) maintains a fingerprint database (602) for storing and queryingfingerprints derived by the media devices.

7.0 Presenting Messages

FIG. 7 illustrates a flow diagram for presenting messages in accordancewith an embodiment. One or more of the steps described below may beomitted, repeated, and/or performed in a different order. Accordingly,the specific arrangement of steps shown in FIG. 7 should not beconstrued as limiting the scope of the invention.

Initially, message preferences associated with a user are received (Step702). Message preferences generally represent any preferences associatedwith message content, message timing, message filtering, messagepriority, message presentation, or any other characteristics associatedwith messages. For example, message preferences may indicate thatmessages are to be presented as soon as they are received or held untila particular time (e.g., when commercials are being displayed). Messagepreferences may indicate different preferences based on a message sourceor a message recipient. For example, messages from a particular website,Really Simply Syndication (RSS) feed, or a particular user may beclassified as high priority messages to be presented first or to bepresented as soon as they are received. Low priority messages may beheld for a particular time. Message preferences may indicate whethermessages are to be presented as received, converted to text, convertedto audio, presented in a particular manner/format/style, etc. Messagepreferences may be associated with automated actions, where receivingparticular messages results in automatically performing specifiedactions. One or more preferences (e.g., message preferences), viewinghistory, and/or other information associated with a user make up a userprofile.

In an embodiment, message preferences may include a user-defined alertcondition. For example, the alert condition may include receiving anemail, voicemail, text message, instant message, twitter tweet, etc.that meets a particular condition. An alert condition may include aspecific user action performed by a specified list of users. Forexample, an alert condition may a particular user posting a hikingactivity invite on a webpage. The alert condition may be based onparticular keywords in a communication, a subject matter associated witha communication, etc. For example, if the word “emergency” or “urgent”is found in the communication, the alert condition may be met. The alertcondition may be related to security (e.g., a house alarm or car alarmbeing set off). The alert condition may be related to kitchen equipment.For example, the alert condition may be linked to an oven timer goingoff. The alert condition may include a change in status of a userspecified entity. For example, the alert condition may be related towhen a user on a social networking website changes status from “in arelationship” to “single”. An alert condition may include theavailability of a particular media content, in a content stream,selected based on a user profile. For example, the user profile mayinclude a viewing history, an actor name, a media content genre, alanguage associated with the media content. If media content thatmatches any part of the user profile, the alert condition may be met andan alert may be presented in response.

In an embodiment, message preferences may be received as direct inputfrom a user, determined based on user files, obtained from the internet(e.g., from a web page or other file associated with a user, by queryinga database, etc.). The message preferences may be obtained by monitoringthe usage patterns on a media device. For example, if usage patternsindicate that a user checks messages immediately upon receivingnotifications of a message, the message preferences may indicate thatmessages are to be displayed or played immediately. Message preferencesfor a user may also be sender based. For example, the sender of amessage may indicate the delivery method and/or delivery preferences.Message preferences may also be randomly (e.g., user input),periodically, or continuously be modified.

In an embodiment, a command to play media content is received (Step704). The received command may be submitted by a user via a keyboard,remote control, a mouse, joystick, a microphone or any other suitableinput device. The command may be a selection in the electronicprogramming guide (EPG) by a user for the playing of the media content.The command may be a channel selection entered by a user. The commandmay be a request to display a slide show of pictures. The command may beto play an audio file. The command may be a request to play a movie(e.g., a command for a blu-ray player). In an embodiment, receiving thecommand to present media content may include a user entering the titleof media content in a search field on a user interface. The command toplay media content may be a user selection of particular media contentthat is stored in memory.

In an embodiment, the media content is played (Step 706). In anembodiment, the media content may be played in response to the commandor without receiving a command. For example, a user may turn on a mediadevice which is automatically configured to receive a content stream onthe last selected channel or a default channel. In an embodiment, themedia device may automatically select media content for playing based onuser preferences or responsive to playing or recording of the mediacontent on another media device.

In an embodiment, a message may be received while playing media content(Step 708). The message may be received from a local or remote sourceover a network (e.g., internet, intranet, broadcast service, etc.). Amessage may be received from a web service through an internetconnection. For example, friend messages or status changes associatedwith a social networking website may be received from a web service. Theweb service may be configured to provide all messages associated with asocial networking website or a filtered selection of messages associatedwith particular preferences. Another example, may include a ReallySimply Syndication (RSS) feed that may be received from a web serviceassociated with news, sports, entertainment, weather, stocks, or anyother suitable category. In an embodiment, the message may be receivedfrom a content source related to services provided by the contentsource. For example, the message may indicate the availability of carpurchasing service, or the availability of a particular car for sale.

The message may be a direct message to a user or group of users (e.g.,voicemail, text message, email, etc.). The message may be received in aform different than the originating form. For example, a text messagemay be received as an audio file, or the text message may be convertedto an audio file by the media device after receipt of the text message.Conversely, an audio file may be received as a text message or convertedto a text message. In an embodiment, symbols, abbreviations, images,etc. may be used to represent messages. In an embodiment, a messagereceived in one language may be translated to a different language.

In an embodiment, the receiving the message may include detecting theoccurrence of a user-defined alert condition. For example, all messagesmay be monitored and compared to user-defined alert conditions. In anembodiment, EPG data, an RSS feed, a webpage, an event log, displayedinformation obtained using OCR or any other source of information may bemonitored for occurrence of the alert condition. If any of the messagesreceived match an alert condition, the occurrence of the alert conditionmay be identified. An alert may be then be immediately presentedindicating occurrence of the alert condition. The message indicatingoccurrence of the alert condition may be interpreted based on userpreferences.

A determination may be made whether to present the message immediately,present the message at a later time, or not present the message at all(Step 710). Based on the user preference, a received message may bepresented (Step 717) immediately upon receiving, or held until a latertime. A message may be presented during commercial breaks, when a userselects the messages for viewing, based on a specified schedule or atanother suitable time. The messages may also be filtered out based onuser preferences. For example, each received message may be compared touser defined alert conditions to determine if the message matches a userdefined alert condition. Messages that match a user defined alertcondition may be presented and messages that do not match the userdefined alert conditions may be filtered out.

In an embodiment, presenting the message may include presenting themessage in a visual format and/or playing the message in an audioformat. For example, a message may be presented by loading a mediacontent frame into a frame buffer and overlaying message content in theframe buffer to overwrite a portion of the media content frame. Thecontent of the frame buffer may then be presented on a display screen.In another exemplary implementation, different buffers may be used formedia content and for message content, where content for the displayscreen is obtained from both buffers. In an embodiment, presenting amessage may include displaying message information and concurrentlyplaying an audio file with the message information. The messageinformation displayed on the screen and played in the audio file may bethe same or different. For example, the display screen may display theface of a person associated with the message or announcing the message,while the audio file may include the actual message. In embodiment,playing an audio message may include muting or lowering the volumeassociated with the media content be played.

8.0 Interpreting Commands

FIG. 8 illustrates a flow diagram for interpreting a voice command inaccordance with an embodiment. One or more of the steps described belowmay be omitted, repeated, and/or performed in a different order.Accordingly, the specific arrangement of steps shown in FIG. 8 shouldnot be construed as limiting the scope of the invention.

Initially, one or more users present near a multimedia device areidentified (Step 802). One or more users may be identified based onvoice input received by the multimedia device or an input device (e.g.,a microphone, a remote) associated with the multimedia device. Forexample, the multimedia device (or an associated input device) may beconfigured to periodically sample detectable voice input and compare thevoice input to data representing user voices to identify known users.The data representing user voices may be generated based on a voicetraining exercise performed by users for the multimedia device toreceive voice samples associated with a user. Users may be identifiedduring an active or passive mode. For example, users may be identifiedwhen a user command is received to recognize users or users may beidentified automatically without a specific user command. Although voiceidentification is used as an example, other means for recognizing usersmay also be used. For example, user names may be entered via an inputdevice (e.g., keyboard, mouse, remote, joystick, etc.). Users may beidentified based on metadata associated with the household. Users may beidentified using fingerprint detection on the media device orfingerprint detection on another communicatively coupled device (e.g., aremote).

In an embodiment, a voice command is received from a user (Step 804). Avoice command may be received by a user first indicating that a voicecommand is to be given. For example, a user may say a keyword such as“command” or enter input on a device such as a remote indicating thatthe user is going to submit a voice command. A voice command may bereceived by continuously processing all voice input and comparing thevoice input to known commands to determine if a voice command wassubmitted. For example, voice input in the last n seconds from thecurrent time may be continuously submitted for analysis to determine ifa voice command was received in the last n seconds. In an embodiment,different portions of the voice command may be received from differentusers. For example, a command “record” may be received from a first userand various titles of programs/shows may be received from multipleusers. Examples of other commands include “order pizza”, “tweet thisgame is amazing”, “wall post who wants to come watch the emmys”, etc.Although a voice command is used in this example, any type of input(e.g., using a mouse, a keyboard, a joystick) may be accepted.

The command may be interpreted based on preferences (e.g., in a userprofile) associated with one or more identified users (Step 806) todetermine an action to be performed (Step 808). Interpreting a commandmay involve determining whether the command is applicable to one user(e.g., the user giving the command) or multiple users (e.g., includingmultiple users identified in Step 802). A particular command word may beindicative of a single user command or a multiple user command. Forexample, tweet commands may be interpreted by default as a commandapplicable to a single user, e.g., the user submitting the command.Furthermore, the command may be interpreted based on the user'spreferences/settings. If the user submitting the command “tweet thisgame is amazing” is associated with a twitter account, then the actionto be performed is to generate a tweet for the user's twitter accountincluding the words “this game is amazing”. Another example of a commandapplicable to a single user includes “wall post who wants to come watchthe emmys”. In this case, the command by a user may be recognized as aFacebook wall post and the message “who wants to come watch the emmys”may be posted on the user's Facebook profile. The multimedia device maybe configured to associate certain types of commands with multiple usercommands. For example, orders for food may be associated with all theidentified users. A command “order pizza” may be interpreted as an orderfor pizza with toppings matching the preferences of all the identifiedusers. A command “buy tickets” may be interpreted as an order topurchase tickets for all the identified users for a football gamecurrently being advertised on television. A command may be intentionallyvague for complete interpretation based on the identified users. Forexample, the command “play recorded show” may result in evaluating eachrecorded show on a media device to determine how many identified usersprefer the recorded show based on user preferences. Thereafter, therecorded show that matches the preferences of the largest number ofidentified users is selected for playing.

In an embodiment, all or a portion of command interpretations may beconfirmed with a user before execution. For example, when orderingpizza, the pizza toppings selected based on user preferences may bepresented for confirmation. Another example involving confirmation ofcommands may involve any orders requiring money or a threshold amount ofmoney.

In an embodiment, a command may be interpreted based on permissionsassociated with a user and the command may be performed only if the usergiving the command has the permission to give the command. For example,a recording and/or playing of a rated R movie may be restricted to usersover the age of seventeen. A profile may be setup for each userincluding the age of the user. If an identified user over the age ofseventeen gives the command to record/play an R rated movie, the commandis executed. However, if a user under the age of seventeen gives thecommand to record/play the R rated movie, the command is denied. In anembodiment, a command may be interpreted based on the religious and/orpolitical beliefs of a user. For example, an election coverage programsponsored by the democratic party may be recorded if a democratic usersubmits a command to record election coverage and an election coverageprogram sponsored by the republican party may be recorded if arepublican user submits the command.

In an embodiment, a language used to submit a command may be used tointerpret the command. For example, if a command to record a show issubmitted in French, the French subtitles may be selected out of a setof available subtitle streams and recorded with the show. In anotherexample, if multiple audio streams are available in different languages,the audio stream selected may be based on the language of the command.

9.0 Correlating Input with Media Content

FIG. 9 illustrates a flow diagram for correlating annotations with mediacontent in accordance with an embodiment. One or more of the stepsdescribed below may be omitted, repeated, and/or performed in adifferent order. Accordingly, the specific arrangement of steps shown inFIG. 9 should not be construed as limiting the scope of the invention.Furthermore, although specific types of annotations (e.g., audio,textual, graphical, etc.) may be discussed in the examples below,embodiments of the invention are applicable to any type of annotation.

In an embodiment, media content is played (Step 902). The media contentmay include both audio and video content, or the media content mayinclude video content alone. Concurrently with playing of the mediacontent, audio input received from a user may be recorded (Step 904).The audio input received from a user may be general reactions to themedia content. For example, the audio input may include laughter,excitement (e.g., gasps, “wow”, etc.), commentary, criticisms, praises,or any other reaction to the media content. In an embodiment, thecommentary may include audio input intended for a subsequent playing ofthe media content. For example, in a documentary film about touristdestinations, a user may submit voice input which includes stories ormemories associated with the particular tourist destination beingfeatured. In another example, a band may provide song lyrics during aparticular portion of the media content for recording in associationwith that portion of the media content. In another embodiment, a usermay provide commentary, plot synopsis, character lines, or any otherinformation about the media content in an alternate language during theplaying of the media content in the original language. Differentversions of audio input (e.g., by the same user or by different users)may be recorded in association with particular media content. In anembodiment, the audio input may be provided with instructions forintended playback information. For example, the playback information mayindicate that the submitted audio is to replace the original audioentirely, or played in concurrently with the original audio. In anembodiment, the audio input may be automatically generated by atext-to-speech translator which generates speech based on textassociated with the media content. For example, speech in an alternatelanguage may be generated based on the closed caption text in thealternate language. In an embodiment, optical character recognition maybe used to identify building names, letters, team names, etc. displayedon a screen and converted to audio for visually impaired audiences, orfor audiences that cannot read the information (e.g., due to languagebarriers or age). In an embodiment, audio input may be receivedconcurrently with playing a particular portion of the media content andstored in association with that particular portion of the media content.

In an embodiment, the media content is subsequently played with theaudio input received during a previous playing of the media content(Step 906). Playing the additional audio input received during theprevious playing of the media content may include completely replacingthe original audio stream or playing concurrently with the originalaudio stream. In an embodiment, the additional audio input may be afeature that can be turned on or off during the playing of thecorresponding media content. In an embodiment, multiple versions ofadditional audio input may be offered, where a user selects theparticular additional audio input for playing during playing of themedia content. For example, an online community may be established forsubmitting and downloading commentary to be played with differentmovies. Different users with different media devices may record audioinput in association with a particular movie (or other content) andthereafter upload the audio input for association with that movie. Whena purchaser of the movie downloads the movie, the purchaser may be ableto select a commentary (e.g., audio input) by another user to bedownloaded/played with the movie. If a purchaser finds the commentary bya particular user hilarious, the purchaser may set the particular useras a default commentator and download all commentaries by the particularuser when downloading a movie (or other media content).

Although audio input is used an example of annotations of media content,any type of annotations may be used in accordance with embodiments ofthe invention. For example, during the playing of media content, textmay be entered or images may be submitted by one or more users. In anembodiment, all or part of an annotation or collection of annotationsmay be processed or analyzed to derive new content. In an embodiment, acollection of annotations associated with the same media content may becompared to identify annotations patterns. For example, a collection ofannotations can be analyzed to determine the most annotated point withinmedia content. Accordingly, a scene or actor which resulted in thegreatest amount of user excitement (or other emotion) may be identifiedvia annotations during a scene. In another example, user contentincluded in a collection of annotations, such as text or voice notes canbe analyzed to determine collective user sentiment (e.g., the funniestscene in a movie, or the funniest movie released in 2009).

10.0 Eliciting Annotations by a Personal Media Device

In an embodiment, any annotations (including audio input, textual input,graphical input, etc.) may be elicited before, during, or afterpresenting media content by a personal media device associated with auser. Eliciting annotations may be based on selections by anadministrator, content producer, content director, etc. For example, auser may be prompted by a media device for a review (e.g., vote, rating,criticism, praise, etc.) at the conclusion of each performance within apresentation of a talent contest within media content in the contentstream that was received by the media device and displayed by the mediadevice. In an embodiment, elicited annotations (or other annotations)may be associated with the media content as a whole rather than aspecific point within the media content such as when the audio input wassubmitted. The annotations of one or more users may then be processed(e.g., to count votes, scores, etc.) for the media content.

In an embodiment, the audio input may be elicited from a user by a mediadevice to build a user profile. For example, reactions to differentmedia content may be elicited from a user. Based on the reactions, auser profile may be automatically created which may include usersinterests, likes, dislikes, values, political views etc. Theautomatically created profile may used for a dating service, a socialnetworking website, etc. The automatically generated profile may bepublished on a webpage (e.g., of a social networking website).

In an embodiment, the system can elicit user annotations to identifyinformation associated with media content. For example, annotations maybe elicited for identification of a face which although detected, cannotbe identified automatically. A system may also be configured to elicitannotations from a parent, after media content has been played,indicating whether the media content is appropriate for children.

11.0 Marking Media Content

In an embodiment, annotations may be used by a user to mark a locationin the playing of media content. For example, a user may submit audioinput or textual input during the playing of media content that includesa particular keyword such as “mark”, “note”, “record”, etc. thatinstructs the system to mark a current location in the playing of themedia content. The system may automatically mark a particular locationbased on user reaction. For example, user input above a certainfrequency or a certain decibel level may indicate that the user isexcited. This excitement point may be stored automatically. In anembodiment, the marked points may include start points and/or endpoints. For example, periods of high user activity which may correlateto exciting portions of a sports game may be marked by start and endpoints. A parent may mark start and end points of media content that arenot appropriate for children and thus, the marked portion may be skippedduring playback unless a password is provided. A user may mark a sectionin a home video that was eventful. As a result of the user marking thepoint or the automatic marking based on user reaction, an annotation maybe stored in association with the point. The annotation may embody areference to the original content, a time, or frame offset from thestart of the original content, and the UTC when the user marked thepoint. Although audio input may used as an example, input may besubmitted by pressing a key on a remote, clicking on a mouse, entering acommand on a keyword, or using any other input method.

In an embodiment, marking (or identifying) a particular point in mediacontent may involve marking a media frame. For example, media frames maybe marked using tags, as described in Applicant owned patent applicationSer. No. 09/665,921 filed on Sep. 20, 2000, which is hereby incorporatedby reference. Another example may involve marking a media frame usinghash values, as described in Applicant owned patent application Ser. No.11/473,543 filed on Jun. 22, 2006, which is hereby incorporated byreference. In an embodiment, marking a particular point in the mediacontent may involve deriving a fingerprint from one or more frames inthe media content and using the fingerprint to recognize the particularpoint in the media content. In an embodiment, a particular point may bemarked by storing a time interval from a starting point in the playingof the media content.

In an embodiment, a user marked location may be selected by the user ata later time. For example, the user may be able to scan throughdifferent user marked locations during the playing of the media contentby pressing next or scan. An image from each of the marked points may bepresented to the user, where the user can select a particular image andstart/resume the playing of the media content from the correspondinguser marked point. User annotations may be used to dynamically segmentmedia content into different parts. User annotations may also be used tofilter out certain portions (e.g., periods of no annotations/excitement)of media content and play the remaining portions of the media content ina subsequent playing of the media content.

12.0 Publication of Media Content Annotations

In an embodiment, all or part of an annotation may be published (e.g.,referenced or presented on a web site or web service). In an embodiment,all or part of an annotation may be automatically presented to a user onanother system. In an example, a user can request the system to send allor parts of annotations to an email or SMS address. In another example,a user can request the system automatically add a movie to an onlineshopping cart or queue when another user (e.g., a movie critic orfriend) positively annotates the movie. In an embodiment, annotations ofmedia content may be sold by a user in an online community for the saleor trade of media content annotations. In an embodiment, annotations(e.g., media content with embedded annotations) may be directed sentfrom one media device to another media device (e.g., through email,intranet, internet, or any other available method of communication).

13.0 Automatically Generated Annotations

In an embodiment, the system can derive annotation content for mediacontent from the closed-captioning portion of the media content. In anexample, the system can produce an annotation that includes a propername recognized by a natural language processing system and/or asemantic analysis system, and then associate the annotation with thevideo content where the proper name appears in closed caption. Inanother example, the system can produce an annotation indicating thestart of a commercial break when the phrase “we'll be back after thesewords” or a similar phrase is recognized in the closed captioning.Another example includes a system producing an annotation associatedwith a region of media content that contains explicit closed captionlanguage. The system may then provide an option to automatically mutethe audio portion of the media content associated with the explicitclosed caption language.

In an embodiment, the system can generate audio input utilizing opticalcharacter recognition systems. In an example, the system can produce anannotation that includes the title of a movie being advertised. Forexample, the annotation may display the movie title (e.g., at the bottomof a screen) as soon as the title of the movie is identified or at theend of a movie trailer. In another example, the system can produce anaudio annotation that includes the names of cast members from videocontent corresponding to credits. Another example may involve the systemproducing an annotation indicating a change in score during a sportsgame by analyzing OCR-derived data inside the ticker regions of asporting event broadcast.

In an example, the system may detect a user is navigating an electronicprogramming guide (EPG) by recognizing a collection of show and movietitles from the OCR. The system may then produce a visual annotation onthe EPG recommending the highest-rated show listed in the EPG. In anembodiment, the annotation may also include other contextual informationthat can be used to further optimize recommendations. For example, theannotation may be based on content recently viewed by the user, whichcan be used to recommend content from the EPG in the same genre orstarring the same actors.

In an embodiment, the system can derive annotation content utilizingspeech-to-text systems. For example, the system can produce a transcriptof the dialogue in media content to be used in a future presentationwhen audio is muted or when requested by the hearing impaired. In anembodiment, the derived transcript can be processed by a separate systemthat monitors presence of topics or persons of interest and thenautomatically produces annotations associated with topics or persons ofinterest.

14.0 Environment Configuration

FIG. 10 shows an exemplary system for configuring an environment inaccordance with one or more embodiments. In an embodiment, theenvironment configuration engine (1015) generally represents anysoftware and/or hardware that may be configured to determine environmentconfigurations (1025). The environment configuration engine (1015) maybe implemented within the media device, shown in FIG. 1B or may beimplemented as a separate component. The environment configurationengine (1015) may identify one or more users (e.g., user A (1005), userN (1010), etc.) that are within close proximity of the environmentconfiguration engine (1015) and identify user preferences (1020)associated with the identified users. The users may be identified basedon voice recognition or based on other input identifying the users.Based on the user preferences (1020), the environment configurationengine may configure a user interface, an audio system configuration, aroom lighting, a game console, a music playlist, a seatingconfiguration, or any other suitable environmental configurations(1025). For example, if five friends are identified, which areassociated with a group user preference, a channel streaming a sportsgame may be automatically selected and surround sound may be selectedfor the audio stream(s) associated with the sports game. Another examplemay involve identifying a couple, and automatically initiating theplaying of a romantic comedy.

15.0 Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 11 is a block diagram that illustrates a System 1100upon which an embodiment of the invention may be implemented. System1100 includes a bus 1102 or other communication mechanism forcommunicating information, and a hardware processor 1104 coupled withbus 1102 for processing information. Hardware processor 1104 may be, forexample, a general purpose microprocessor.

System 1100 also includes a main memory 1106, such as a random accessmemory (RAM) or other dynamic storage device, coupled to bus 1102 forstoring information and instructions to be executed by processor 1104.Main memory 1106 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 1104. Such instructions, when stored in storagemedia accessible to processor 1104, render System 1100 into aspecial-purpose machine that is customized to perform the operationsspecified in the instructions.

System 1100 further includes a read only memory (ROM) 1108 or otherstatic storage device coupled to bus 1102 for storing static informationand instructions for processor 1104. A storage device 1110, such as amagnetic disk or optical disk, is provided and coupled to bus 1102 forstoring information and instructions.

System 1100 may be coupled via bus 1102 to a display 1112, such as acathode ray tube (CRT), for displaying information to a computer user.An input device 1114, including alphanumeric and other keys, is coupledto bus 1102 for communicating information and command selections toprocessor 1104. Another type of user input device is cursor control11111, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1104 and for controlling cursor movement on display 1112. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

System 1100 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the System causes or programsSystem 1100 to be a special-purpose machine. According to oneembodiment, the techniques herein are performed by System 1100 inresponse to processor 1104 executing one or more sequences of one ormore instructions contained in main memory 1106. Such instructions maybe read into main memory 1106 from another storage medium, such asstorage device 1110. Execution of the sequences of instructionscontained in main memory 1106 causes processor 1104 to perform theprocess steps described herein. In alternative embodiments, hard-wiredcircuitry may be used in place of or in combination with softwareinstructions.

The term “storage media” as used herein refers to any media that storedata and/or instructions that cause a machine to operation in a specificfashion. Such storage media may comprise non-volatile media and/orvolatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as storage device 1110. Volatile media includesdynamic memory, such as main memory 1106. Common forms of storage mediainclude, for example, a floppy disk, a flexible disk, hard disk, solidstate drive, magnetic tape, or any other magnetic data storage medium, aCD-ROM, any other optical data storage medium, any physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, anyother memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1102. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1104 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to System 1100 can receivethe data on the telephone line and use an infra-red transmitter toconvert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1102. Bus 1102 carries the data tomain memory 1106, from which processor 1104 retrieves and executes theinstructions. The instructions received by main memory 1106 mayoptionally be stored on storage device 1110 either before or afterexecution by processor 1104.

System 1100 also includes a communication interface 1118 coupled to bus1102. Communication interface 1118 provides a two-way data communicationcoupling to a network link 1120 that is connected to a local network1122. For example, communication interface 1118 may be an integratedservices digital network (ISDN) card, cable modem, satellite modem, or amodem to provide a data communication connection to a corresponding typeof telephone line. As another example, communication interface 1118 maybe a local area network (LAN) card to provide a data communicationconnection to a compatible LAN. Wireless links may also be implemented.In any such implementation, communication interface 1118 sends andreceives electrical, electromagnetic or optical signals that carrydigital data streams representing various types of information.

Network link 1120 typically provides data communication through one ormore networks to other data devices. For example, network link 1120 mayprovide a connection through local network 1122 to a host computer 1124or to data equipment operated by an Internet Service Provider (ISP)11211. ISP 11211 in turn provides data communication services throughthe world wide packet data communication network now commonly referredto as the “Internet” 1128. Local network 1122 and Internet 1128 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1120 and through communication interface 1118, which carrythe digital data to and from System 1100, are example forms oftransmission media.

System 1100 can send messages and receive data, including program code,through the network(s), network link 1120 and communication interface1118. In the Internet example, a server 1130 might transmit a requestedcode for an application program through Internet 1128, ISP 11211, localnetwork 1122 and communication interface 1118.

The received code may be executed by processor 1104 as it is received,and/or stored in storage device 1110, or other non-volatile storage forlater execution. In an embodiment, an apparatus is a combination of oneor more hardware and/or software components described herein. In anembodiment, a subsystem for performing a step is a combination of one ormore hardware and/or software components that may be configured toperform the step.

16.0 Extensions and Alternatives

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method comprising: playing media content on a first multimediadevice; concurrently with playing a particular portion of the mediacontent on the first multimedia device, receiving a first audio inputfrom a first user; storing the first audio input in association with theparticular portion of the media content.
 2. The method as recited inclaim 1, further comprising: playing the particular portion of the mediacontent concurrently with playing the first audio input received fromthe user.
 3. The method as recited in claim 1, further comprising:playing the media content on a second multimedia device; concurrentlywith playing the particular portion of the media content on the secondmultimedia device, receiving a second audio input from a second user;storing the second audio input in association with the particularportion of the media content.
 4. The method as recited in claim 3,further comprising: the first media device uploading the first audioinput to a web server; the second media device uploading the secondaudio input to a web server; a third media device selecting one of thefirst audio input or the second audio input for downloading with themedia content; the third media device downloading the media content withthe first audio input or the second audio input.
 5. The method asrecited in claim 1, further comprising: identifying text informationassociated with the media content based on optical character recognitionby applied to the particular portion of the media content; generating anaudio file based on the text information; storing the audio file inassociation with the particular portion of the media content.
 6. Themethod as recited in claim 1, further comprising: receiving a secondaudio input concurrently with playing the media content; analyzing atleast the first audio input and the second audio input to identify apattern associated with the media content.
 7. The method as recited inclaim 1, further comprising generating a user profile for the first userbased at least on the first audio input received from the first user. 8.A computer readable storage medium comprising a sequence ofinstructions, which when executed by one or more processors, causeperforming steps comprising: playing media content on a first multimediadevice; concurrently with playing a particular portion of the mediacontent on the first multimedia device, receiving a first audio inputfrom a first user; storing the first audio input in association with theparticular portion of the media content.
 9. The computer readablestorage medium as recited in claim 8, the steps further comprising:playing the particular portion of the media content concurrently withplaying the first audio input received from the user.
 10. The computerreadable storage medium as recited in claim 8, the steps furthercomprising: playing the media content on a second multimedia device;concurrently with playing the particular portion of the media content onthe second multimedia device, receiving a second audio input from asecond user; storing the second audio input in association with theparticular portion of the media content.
 11. The computer readablestorage medium as recited in claim 10, the steps further comprising: thefirst media device uploading the first audio input to a web server; thesecond media device uploading the second audio input to a web server; athird media device selecting one of the first audio input or the secondaudio input for downloading with the media content; the third mediadevice downloading the media content with the first audio input or thesecond audio input.
 12. The computer readable storage medium as recitedin claim 8, the steps further comprising: identifying text informationassociated with the media content based on optical character recognitionby applied to the particular portion of the media content; generating anaudio file based on the text information; storing the audio file inassociation with the particular portion of the media content.
 13. Thecomputer readable storage medium as recited in claim 8, the stepsfurther comprising: receiving a second audio input concurrently withplaying the media content; analyzing at least the first audio input andthe second audio input to identify a pattern associated with the mediacontent.
 14. The computer readable storage medium as recited in claim 8,the steps further comprising generating a user profile for the firstuser based at least on the first audio input received from the firstuser.
 15. An apparatus comprising: one or more processors; one or moresubsystems configured for performing steps comprising: playing mediacontent on a first multimedia device; concurrently with playing aparticular portion of the media content on the first multimedia device,receiving a first audio input from a first user; storing the first audioinput in association with the particular portion of the media content.16. The apparatus as recited in claim 15, the steps further comprising:playing the particular portion of the media content concurrently withplaying the first audio input received from the user.
 17. The apparatusas recited in claim 15, the steps further comprising: playing the mediacontent on a second multimedia device; concurrently with playing theparticular portion of the media content on the second multimedia device,receiving a second audio input from a second user; storing the secondaudio input in association with the particular portion of the mediacontent.
 18. The apparatus as recited in claim 10, the steps furthercomprising: the first media device uploading the first audio input to aweb server; the second media device uploading the second audio input toa web server; a third media device selecting one of the first audioinput or the second audio input for downloading with the media content;the third media device downloading the media content with the firstaudio input or the second audio input.
 19. The apparatus as recited inclaim 15, the steps further comprising: identifying text informationassociated with the media content based on optical character recognitionby applied to the particular portion of the media content; generating anaudio file based on the text information; storing the audio file inassociation with the particular portion of the media content.
 20. Theapparatus as recited in claim 15, the steps further comprising:receiving a second audio input concurrently with playing the mediacontent; analyzing at least the first audio input and the second audioinput to identify a pattern associated with the media content.
 21. Theapparatus as recited in claim 15, the steps further comprisinggenerating a user profile for the first user based at least on the firstaudio input received from the first user.